Zeroth-Order Nonconvex Stochastic Optimization: Handling Constraints, High Dimensionality, and Saddle Points

In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex convex optimization, with a focus on addressing constrained high-dimensional setting, saddle point avoiding. To handle first generalizations of the conditional gradient algorithm achieving rates similar to standard using only information. facilitate optimization in high dimensions, explore advantages structural sparsity assumptions. Specifically, (i) highlight an implicit regularization phenomenon where information adapts problem at hand by just varying step size (ii) truncated information, whose rate convergence depends poly-logarithmically dimensionality. We next avoiding points setting. Toward that, interpret Gaussian smoothing technique estimating based as instantiation first-order Stein’s identity. Based this, provide novel linear-(in dimension) time estimator Hessian matrix function which is second-order then variant cubic regularized Newton method discuss its local minima.